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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Application No.: Not yet assigned - 371 of PCT/J POO/06393 

Filing Date: Not yet assigned 

Applicant: MATSUSHITA ELECTRIC INDUSTRIAL CO., LTD, et al. 

Title: DRIVING SUPPORT SYSTEM 

Attorney Docket: 5077-000077 

Box PCT 

Hon. Commissioner of Patents and Trademarks 
[J Washington, D.C. 20231 

Jjj PRELIMINARY AMENDMENT 

ffi Sir: 

y Applicant herewith submits this Preliminary Amendment to the application filed 

O herewith, for consideration prior to the calculation of the filing fee, as follows: 

IN THE CLAIMS 



Please amend Claims 1 in accordance with the following rewritten claims in clean 
form. Please cancel Claim 2 without prejudice or disclaimer of the subject matter 
contained therein. Applicant includes herewith an Attachment for Claim Amendments 
showing a marked up version of each amended claim. 

1. (AMENDED) A system for supporting driving of a moving body 
comprising: 

a plurality of imaging means installed on said moving body for taking camera 
images of the rear of said moving body; and 



detection means for detecting movement of an object present in the rear of said 
moving body based on said camera images of said plurality of imaging means, 

wherein said plurality of imaging means have, in camera images thereof, an 
overlap area that includes an area in the vicinity of a vanishing point and in which 
imaging areas of first imaging means and second imaging means overlap with each 
other, 

said detection means obtains a stereo parallax between said first imaging means 
and said second imaging means in said overlap area, and obtains a distance to said 
object on the basis of said obtained stereo parallax, and 

wherein said detection means detects flows corresponding to movement with 
time of an image in the imaging area of said first imaging means excluding said overlap 
area, and detects movement of said object present in the rear of said moving body on 
the basis of said detected flows. 

2. (Cancelled) 
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REMARKS 



Claim 1 has been amended. Claim 2 has been cancelled. Claims 3-9 



remain unchanged. After acceptance of this amendment, Claimsl and 3-9 will remain 
pending in the current application. Applicants respectfully submit that no new matter has 
been added and that the basis for the amendments can be found throughout the 
drawings, specification, and claims as originally filed. Consideration of the application as 
amended is requested. 



Respectfully submitted, 



Date: 19 March 2002 




Gregory A>Stobbs, Reg. No. 28764 
Attorney for Applicant 
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ATTACHMENT FOR CLAIM AMENDMENTS 

The following is a marked up version of each amended claim in which 
underlines indicates insertions and brackets indicate deletions. 

1. (AMENDED) A system for supporting driving of a moving body 
comprising: 

a plurality of imaging means installed on said moving body for taking camera 
images of the rear of said moving body; and 

detection means for detecting movement of an object present in the rear of said 
moving body based on said camera images of said plurality of imaging means, 

wherein said plurality of imaging means have, in camera images thereof, an 
overlap area that includes an area in the vicinity of a vanishing point and in which 
imaging areas of first imaging means and second imaging means overlap with each 
other, [and] 

said detection means obtains a stereo parallax between said first imaging means 
and said second imaging means in said overlap area, and obtains a distance to said 
object on the basis of said obtained stereo parallax, and 

wherein said detection means detects flows corresponding to movement with 
time of an image in the imaging area of said first imaging means excluding said overlap 
area, and detects movement of said object present in the rear of said moving body on 
the basis of said detected flows. 



2. (Cancelled) 
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Description 
DRIVING SUPPORT SYSTEM 

Technical Field 

The present invention relates to a driving support 
system for supporting driving of a moving body such as a 
vehicle by taking an image of environmental conditions of the 
moving body with a camera and processing camera imagery. 

Background Art 

In respect of a conventional driving support system 
using a camera, detection of an obstacle or an approaching 
vehicle is disclosed in, for example, Japanese Laid-Open 
Patent Publication Nos . 9-240397 and 7-93693. 

Japanese Laid-Open Patent Publication No. 9-240397 
(hereinafter referred to as "the first conventional example") 
discloses an annunciator for a vehicle running at the side 
rear. This annunciator detects, on the basis of a camera 
image taken in the side rear region of the self -vehicle, a 
moving body present in an adjacent lane and also detects 
whether or not a white line is present in detecting a moving 
body, so as to detect another vehicle by integrating the 
results of these detection. Thus, closeness between the 
self -vehicle and another vehicle is judged, so that a driver 
can be informed when there is a possibility of excessive 



closeness . 

Alternatively, Japanese Laid-Open Patent Publication No. 
7-93 693 (hereinafter referred to as "the second conventional 
example") discloses an object detector for a vehicle* This 
is an object detector for a vehicle capable of correctly 
distinguishing an object such as a vehicle from a figure or 
pattern drawn on the road so as to accurately detect the 
object. As a characteristic of the structure, the movement 
of an edge point of an object in an image is measured by 
apparent travel speed measuring means as if it were three- 
dimensional movement on the road and the measured movement 
and the speed of the vehicle is compared by object judgment 
means so as to discriminate the object. 

Problems to be Solved by the Invention 

However, the aforementioned conventional techniques 
have the following problems: 

First, in the second conventional example, the method 
for detecting an obstacle through movement analysis of a 
camera image is a method generally designated as motion 
stereo, in which change of an image in accordance with change 
of a visual point through movement is analyzed so as to 
obtain three-dimensional information of the camera image. 
This method, however, has a problem that the change of the 
image in accordance with the change of the visual point is 



small in a camera image taken in a moving direction- when 
this method is applied to, for example, a vehicle, there 
arises a problem that the sensitivity of the detection is 
lowered when an object is closer to the forward or backward 
region along the moving direction. 

Furthermore, since the technique of movement analysis 
is used for obtaining the movement of an object on a screen, 
when the imaging device is installed on a vehicle, the screen 
is vibrated due to jolt of the vehicle, and hence, the 
movement cannot be accurately obtained. 

Moreover, when an approaching object is detected, it is 
significant how the driver is informed of the approaching 
object. In the case where an alarm sound or the like is used, 
the alarm sound may surprise the driver, which may cause a 
mistake in the driving. Also, error detection of an 
approaching object may obstacle safe driving of the driver. 
Thus, it is a significant problem how the driver is informed 
of an approaching object. In respect to a warning to the 
driver, the first conventional example merely describes that 
the driver is informed of excessive closeness. In the second 
conventional example, no description is made on the method 
for informing the driver of a detected obstacle. 



Disclosure of the Invention 

In consideration of such problems of the conventional 



driving support system or moving body image display system, 
an object of the invention is providing a driving support 
system in which a driver can directly confirm environmental 
conditions such as an approaching object and an obstacle so 
5 as to reduce burdens of the driver. 

Specifically, the system for supporting driving of a 
moving body of this invention comprises a plurality of 
imaging means installed on the moving body for taking camera 
images of the rear of the moving body; and detection means 

10 for detecting movement of an object present in the rear of 
the moving body based on the camera images of the plurality 
of imaging means, and the plurality of imaging means have, in 
camera images thereof, an overlap area that includes an area 
in the vicinity of a vanishing point and in which imaging 

15 areas of first imaging means and second imaging means overlap 
with each other, and the detection means obtains a stereo 
parallax between the first imaging means and the second 
imaging means in the overlap area, and obtains a distance to 
the object on the basis of the obtained stereo parallax. 

20 The detection means of the driving support system of 

this invention preferably detects flows corresponding to 
movement with time of an image in the imaging area of the 
first imaging means excluding the overlap area, and detects 
movement of the object present in the rear of the moving- body 

25 on the basis of the detected flows. 

4 



The driving supporting system of this invention 
preferably further comprises image synthesizing means for 
generating an image representing the rear of the moving body 
through image synthesis using the camera images of the 
plurality of imaging means. 

Also the driving supporting system of this invention 
preferably further comprises danger level determining means 
for determining a possibility of collision of the moving body 
with an approaching object from the rear of the moving body 
on the basis of information output from the detection means 
and for outputting an indicator signal when it is determined 
that there is a strong possibility of the collision; and 
external warning means for providing a warning to the rear of 
the moving body when the indicator signal is output from the 
danger level determining means. Alternatively, it preferably 
comprises danger level determining means for determining a 
possibility of collision of the moving body with an 
approaching object from the rear of the moving body on the 
basis of information output from the detection means and for 
outputting an indicator signal when it is determined that 
there is a strong possibility of the collision; and passenger 
protecting means for taking measure to protect a passenger of 
the moving body when the indicator signal is output from the 
danger level determining means . 

Moreover, another system for supporting driving of a 



moving body of this invention comprises imaging means 
installed on the moving body for taking a camera image of a 
surrounding region of the moving body; image generating means 
for converting the camera image of the imaging means into an 
image seen from a visual point positioned differently from 
the imaging means; and detecting means for detecting a 
distance from the moving body to an object imaged in the 
camera image, and the image generating means corrects 
distortion of an image of the object by using the distance 
detected by the detection means in converting the image. 

Preferably, in this driving supporting system, the 
imaging means is plural in number, the plural imaging means 
have, in camera images thereof, an overlap area in which 
imaging areas of first imaging means and second imaging means 
overlap with each other, and the detection means obtains a 
stereo parallax between the first imaging means and the 
second imaging means in the overlap area, and obtains a 
distance to the object on the basis of the obtained stereo 
parallax. 

Furthermore, the detection means of this driving 
support system preferably obtains a distance to the object on 
the basis of flows corresponding to movement with time of the 
camera image. 

Still another system for supporting driving of a moving 
body of this invention comprises imaging means installed on 



the moving body for taking a camera image of a surrounding 
region of the moving body; and detection means for obtaining 
flows corresponding to movement with time on the basis of the 
camera image of the imaging means and for detecting movement 

5 of an object present in the surrounding region of the moving 
body on the basis of the flows, and the detection means 
obtains, as preparation for detecting the movement of the 
object, an offset estimated value from each of the obtained 
flows and cancels the offset estimated value from each of the 

10 flows as a vibration component derived from jolt of the 
moving body. 

Brief Description of Drawings 
FIG. 1 is a block diagram for showing the structure of 
a driving support system according to Embodiment 1 of the 

15 invention. , . 

FIG. 2A is a diagram of an exemplified camera image and 
FIG. 2B is a diagram in which flows are shown on the image of 
FIG. 2A. 

FIG. 3 is a conceptual diagram seen from above of 
20 conditions under which the image of FIG. 2A is taken. 

FIG. 4A is a diagram for showing the relationship 
between a vanishing point and flows and FIG. 4B is a diagram 
of a moving body area extracted from the image of FIG. 2A. 

FIG. 5 is a flowchart for showing procedures of 
25 vibration component extraction. 

7 



FIGS. 6A and 6B are diagrams of influence of vertical 
vibration of a vehicle on imaging means. 

FIGS. 7A through 7G are diagrams for explaining 
procedures for obtaining an offset estimated value of a 
motion vector. 

FIG. 8 is a flowchart for showing procedures in moving 
body/detected object detection. 

FIGS. 9A and 9B are diagrams for explaining a method 
for discriminating a moving body. 

FIGS. 10A and 10B are diagrams for showing extraction 
of a moving body area by using a motion vector. 

FIG. 11A is a diagram of a default distance value of an 
imaging range and FIG. 11B is a diagram for showing distance 
estimation employed when a moving body area extends over 
areas AR1 and AR2. 

FIG. 12A is a diagram of an image where motion vectors 
of still background are shown and FIG. 12B is a diagram for 
explaining judgment of an obstacle. 

FIG. 13 is a diagram of an example of a display image. 

FIG. 14 is a block diagram for showing the structure of 
a driving support system according to Embodiment 2 of the 
invention. 

FIG. 15 is a schematic diagram for showing an example 
of arrangement of imaging means in Embodiment 2 of the 
invention. 

8 
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FIGS, 16A and 16B are diagrams of camera images in 
which flows are shown and FIG. 16C is a diagram in which a 
stereo parallax is obtained by overlapping the two images of 
FIGS, 16A and 16B, 
5 FIG. 17 is a diagram of an example of an image in which 

an obstacle, a moving body and an approaching object are 
extracted. 

FIGS . 18A and 18B are diagrams for showing a virtual 
visual point used for obtaining a synthesized image where a 
10 moving body is three-dimensionally displayed. 

FIG. 19A is a diagram of an example of an actually 
taken camera image and FIG. 19B is a synthesized image 
generated from FIG. 19A. 

FIGS. 20A and 20B are diagrams for explaining image 
15 synthesis in consideration of a detected distance. 

FIGS. 21A and 21B are diagrams for explaining image 
synthesis in consideration of a detected distance by using 
two imaging means. 

FIGS. 22A and 22B are diagrams for showing an example 
20 of arrangement of imaging means according to a modification 
of Embodiment 2 of the invention. 

FIGS. 23A, 23B and 23C are diagrams for explaining a 
method for obtaining a stereo parallax according to the 
modification of Embodiment 2 of the invention. 
25 FIG. 24 is a block diagram for showing the structure of 
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a driving support system according to Embodiment 3 of the 
invention. 

Best Mode for Carrying Out the Invention 

5 Preferred embodiments of the invention will now be 

described with reference to the accompanying drawings, 
(EMBODIMENT 1) 

FIG. 1 is a block diagram for showing the structure of 
a driving support system (moving body image display system) 

10 according to Embodiment 1 of the invention. In FIG. 1, a 
reference numeral 101 denotes imaging means provided on a 
moving body, a reference numeral 102 denotes hierarchical 
image generating means, a reference numeral 103 denotes an 
LPF (low-pass filter), a reference numeral 104 denotes block 

15 sampling means, a reference numeral 105 denotes hierarchical 
block matching means, a reference numeral 106 denotes sub- 
pixel estimating/reliability determining means , a reference 
numeral 107 denotes vibration component extracting/canceling 
means, a reference numeral 108 denotes vanishing point 

20 calculating means, a reference numeral 109 denotes moving 
body/approaching object detecting means, a reference numeral 

110 denotes image synthesizing means, and a reference numeral 

111 denotes a display device. The composing elements 102 
through 109 together form detection means . 

25 The imaging means 101 is, for example, a camera, and is 

10 
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provided on a rear portion, for example, on a rear panel of a 
vehicle so that an image of the rear of the vehicle can be 
taken. As the display device 111, a display dedicated to 
this system or a liquid crystal monitor for car navigation 

5 may be used. 

The object of the moving body image display system of 
this embodiment is mainly reducing burdens of a driver by 
directly and comprehensively warning and displaying an object 
approaching from the rear that can be a danger in changing 
10 the driving course. The operation of the moving body image 
display system of this embodiment will now be described with 
reference to FIGS. 2 through 13. 

First, the imaging means 101 takes an image (herein of 
320 pixels x 240 pixels) of the rear of the self-vehicle as 
15 shown in FIG. 2A. This camera image includes obstacles such 
as a building 11 and a roadside tree 12 and moving bodies 
such as other vehicles 13 and 14 running behind. In FIG. 2, 
these obstacles and moving bodies are schematically expressed. 
This camera image is input to the hierarchical image 
20 generating means 102 and the image synthesizing means 110. 

The hierarchical image generating means 102 generates a 
primary upper image (of 160 x 120 pixels) by adding each of 2 
x 2 pixels of the input image. Furthermore, a secondary 
upper image (of 80 x 60 pixels) is similarly generated from 
25 the primary upper image. These three kinds of images are 

11 



output to the LPF 103 as hierarchical images. The LPF 103 
conducts LPF processing for 3x3 pixels on each of the input 
hierarchical images. 

The block sampling means 104 divides each of the 
5 hierarchical images into blocks each of 8 x 8 pixels, and 
sub-samples each block at 16 (=4x4) representing points 
selected every 2 pixels. 

The hierarchical block matching means 105 obtains, from 
the image of the upper hierarchy, an SAD (a sum of absolute 
10 differences) through block matching in a range of 5 x 5 
pixels, and further obtains a motion vector on the basis of a 
point where the SAD is minimum. With respect to a block of 
the image of the lower hierarchy, a motion vector is further 
obtained in a range of 5 x 5 pixels with a motion vector 
15 obtained in a block of the higher hierarchy disposed in the 
same position as the center. 

The sub-pixel estimating/reliability determining means 
106 estimates, by using the motion vector obtained in the 
image of the lowest hierarchy (camera image) and the SAD, a 
20 motion vector at accuracy of a sub-pixel with one or less 
pixel on the basis ' of values of the SAD at the SAD minimum 
point and eight points around the SAD minimum point. Also, 
the sub-pixel estimating/reliability determining means 106 
determines the reliability of the motion vector in the block. 
25 Through these processing by the hierarchical block 

12 



matching means 105 and the sub-pixel estimating/reliability 
determining means 106, motion vectors corresponding to 
movement from a previous frame at respective points in the 
camera image are obtained as flows FL as shown in FIG. 2B. 
5 The flows FL in the camera image will now be described 

with reference to FIG. 3. As shown in FIG. 2B, the 
aforementioned flows FL are obtained at edge portions of the 
image. At this point, after canceling the vibration of the 
camera, a direction reverse to the running direction of the 

10 vehicle of FIG. 3 corresponds to a vanishing point VP on the 
screen shown in FIG. 4A. An object standing still on the 
ground has a flow FL directing toward the vanishing point VP 
on the screen. Accordingly, an area on the screen having a 
flow FL not directing to the vanishing point VP (such as a 

15 rectangular area 202) can be extracted as a moving body or an 
approaching object. 

Furthermore, the vibration component 

extracting/canceling means 107 extracts and cancels a 
vibration component caused by the jolt of the vehicle by 

20 integrally processing the obtained motion vectors. In this 
invention, since a vibration component of the image is 
extracted by using the motion vectors obtained for detecting 
an approaching object, there is no need to separately detect 
a vibration component of the entire image for vibration 

25 correction. The vanishing point calculating means 108 
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obtains the vanishing point VP of the flows FL of the image 
accompanied by the movement of the car* Specifically, as 
shown in FIG. 4A, the vanishing point VP is obtained as a 
point to which most of the points on the entire image direct. 
5 The moving body /approaching object detecting means 109 

extracts, as a moving body /approaching object candidate block, 
a block having a motion vector different from the flow FL 
directing to the vanishing point VP obtained by the vanishing 
point calculating means 108. Furthermore, moving 

10 body/approaching object candidate blocks close to one another 
are linked, so that an area where the moving body /approaching 
object is present like a rectangular area 202 can be 
extracted as a moving body as shown in FIG. 4B. 

Now, the specific operation of the vibration component 

15 extracting/canceling means 107 will be described with 
reference to FIGS. 5 through 7. FIG. 5 is a flowchart for 
explaining the operation. 

A vehicle not only moves in the running direction but 
also jolts in the vertical direction owing to the influence 

20 of bumps of the road and the like. As shown in FIG. 6, the 
influence of the vertical jolt of the vehicle on the imaging 
means appears as a positional change as shown in FIG. 6A and 
an imaging direction change as shown in FIG. 6B. 

The vertical positional change as shown in FIG . 6A is 

25 very small as compared with a positional change along the 
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running direction of the vehicle because the frame gap of an 
image signal is very small. Also, a change of the position 
of the imaging means is largely varied in its influence 
depending upon the distance to an object, and largely affects 
5 a close object but minimally affects a far object. The 
distance to a rear object monitored in this embodiment is 
several meters through several tens meters, which is 
sufficiently far. Accordingly, in this case, the imaging 
direction change shown in FIG. 6B alone is considered without 

10 considering the influence of the vertical positional change. 

The imaging direction change is not varied in its 
influence depending upon the distance to an object, and when 
the angle of the change is very small, it can be assumed as a 
vertical offset Vdy, which is constant in the entire screen, 

15 with respect to each motion vector on the screen. In this 
case, a motion vector (Vx, Vy) in still background excluding 
a moving body can be approximated to a sum of a motion vector 
(VOx, VOy) directed to the vanishing point VP derived from 
the movement of the vehicle and the offset (0, Vdy) as shown 

20 in the following formulas: 
Vx = VOx 
Vy = VOy + Vdy 

In the flowchart of FIG. 5, the offset (0, Vdy) is 
extracted as the vibration component, so as to be cancelled 
25 from the detected motion vector (Vx, Vy) . Thus, the motion 
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vector (VOx, VOy) directed toward the vanishing point VP is 
obtained in the still background portion. 

First, with respect to each point (x, y) on the screen, 
a motion vector (Vx, Vy) is input (Sll). At this point, with 
5 respect to a uniform area where there is no figure, such as 
the sky, motion vectors are previously excluded because of 
lack of the reliability thereof, and motion vectors with 
reliability alone are input. Also, a virtual vanishing point 
(xO, yO) obtained by assuming that the vehicle runs straight 
10 forward can be previously obtained because it can be 
determined on a predetermined point on the screen on the 
basis of the installation angle of the imaging means 101 
(S12) . 

Next, with respect to each motion vector, the offset is 
15 calculated (S13). In the case where the point (x, y) is 
positioned in the still background portion, the following 
relationships hold : 

(x - xO) * VOy = (y - yO) * Vx 
(x - xO) * (Vy - Vdy) = (y - yO) * Vx 
20 On the basis of these relationships, Vdy is obtained as 
follows: 

Vdy = Vy - (y - yO) * Vx / (x - xO) 

On the basis of this formula, Vdy can be obtained from 
one motion vector. However, the input motion vectors include 
25 a large number of motion vectors with regard to image areas 
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apart form' the still background portion such as a moving body. 
Also, a motion vector with regard to the still background 
portion includes an error. Therefore , the offset Vdy of the 
motion vector is estimated through statistical processing. 
5 Specifically, as shown in FIG. 7, with respect to each motion 
vector, Vdy is obtained in accordance with the aforementioned 
formula, and the frequency of each Vdy is calculated, so that 
Vdy with the highest frequency can be determined as an 
ultimate offset estimated value (S14, S15 and S16). 

10 First, an error (±Vnx, ±Vny) is assumed with respect to 

one motion vector, and a small error (±nx, ±ny) is assumed 
with respect to the position on the screen. It is herein 
assumed that the offset Vdy is obtained with respect to one 
motion vector as shown in FIG. 7A. When the error (±Vnx, 

15 ±Vny) of the motion vector is considered with respect to this 
offset, distributions as shown in FIGS. 7B and 7C are 
obtained, and when the small error (±nx, ±ny) of the position 
on the screen is considered, distributions as shown in FIGS. 
7D and 7E are obtained. Through convolution integral of the 

20 distributions of FIGS. 7B through 7E, a distribution as shown 
in FIG. 7F can be obtained. 

As a sum of products of distributions like that of FIG. 
7F of the motion vectors on the entire screen, a distribution 
as shown in FIG. 7G is obtained. In this distribution, a 

25 value of Vdy with the highest frequency is determined as an 
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estimated value of the offset derived from the vibration. 

The vanishing point calculating means 108 of FIG* 1 
cancels the estimated offset and then obtains an actual 
vanishing point again. At this point, in the case where, for 
example, a moving body occupies a large area in the screen or 
the vehicle is jolted in a direction other than the vertical 
direction or in the case where the vehicle is not running 
straight but making a turn or the like, the calculated 
vanishing point is largely different from the virtual 
vanishing point. In such cases, the subsequent moving 
body/approaching object detecting processing is suspended, 
and an image is synthesized by using a result obtained from a 
previous frame and the synthesized image is displayed. 

On the other hand, in the case where the calculated 
vanishing point falls within a predetermined range in the 
vicinity of the virtual vanishing point, the processing is 
determined to be normal, and the moving body/approaching 
object detecting means 109 conducts the following operation: 

FIG. 8 is a flowchart for showing the specific 
operation of the moving body /approaching object detecting 
means 109. First, in the same manner as in the vibration 
component extraction, a motion vector (Vx, Vy) with respect 
to each point (x, y) on the screen and the virtual vanishing 
point (xO, yO) are input (S21 and S22). 

It is determined whether or not the point is a moving 
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body depending upon whether or not the input vector 
represents movement toward the vanishing point after 
canceling the offset (S23). Specifically, as shown in FIG. 9, 
this determination is made depending upon whether or not the 
5 distribution of Vdy (of FIG. 7F) obtained in the vibration 
component extraction in consideration of the error (±Vnx, 
±Vny) of the motion vector and the small error (±nx, ±ny) of 
the position on the screen accords with the offset estimated 
value. In other words f in FIG. 9A, since the distribution of 

10 Vdy and the offset estimated value do not accord with each 
other, the point of this motion vector is determined as a 
moving body. On the other hand, in FIG. 9B, since the 
distribution of Vdy and the offset estimated value accords 
with each other, the point of this motion vector is 

15 determined as the still background. 

As shown in FIG. 10A, motion vectors FL1 each 
determined as a moving body are detected in respective 
portions of the moving body on the screen. Therefore, an 
area including these motion vectors FL1 is grouped as shown 

20 in FIG. 10B, so as to generate a rectangular moving body area 
202 (S24). A distance from the vehicle to this moving body 
is then estimated on the position of the lower end UE of the 
moving body area 202. 

FIG. 11A is a diagram for showing a default distance 

25 value in an imaging range of the imaging means 101. Herein, 
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as shown in FIG, 11A, a first area ARl below a predetermined 
position in the camera image is assumed to be present on a 
road surface RS, and a second area AR2 above the 
predetermined position is assumed to be away from the imaging 
5 means 101 by a predetermined distance DD (of, for example, 50 
m) . Specifically, with respect to the first area ARl, an 
estimated distance to the road surface RS is stored as the 
default distance value, and with respect to the second area 

y. AR2, the predetermined distance DD is stored as the default 

O 

Q 10 distance value, 

!2 Furthermore, as shown in FIG. 11B, in the case where, 

fit 

for example, the moving body area 202 extends over the areas 
ARl and AR2 on the screen, the distance is estimated on the 
basis of the lower end UE of the moving body area 202 
15 included in the first area ARl by assuming that the object is 
in contact with the road surface RS at the lower end UE. 

The distance to the moving body area estimated at this 
point is stored in a memory. When a moving body area is 
detected in the same position through processing of a 
20 subsequent frame image and the estimated distance to the 
moving body area is shorter than the estimated distance 
obtained in the previous frame and stored in the memory, the 
object included in the moving body area is determined as an 
approaching object (S26). 
25 On the other hand, with respect to motion vectors FL2 
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determined as the still background as shown in FIG. 12A, a 
distance Z is calculated on the basis of the size of the 
vector (with the offset canceled) by the following formula 
(S27): 

5 z = dZ * r / dr 

wherein dZ is a travel length of the vehicle between the 
frames, r is a distance from the vanishing point VP on the 
screen and dr is the size of the motion vector, which are 
represented as follows: 

10 r = sqrt ( (x - xO) 2 + (y - yO) 2 )) 

dr = sqrt (Vx 2 + (Vy - vdy) 2 ) 

The distance Z obtained at this point is compared with 
the distance to the road surface stored as the default 
distance value (S28). Thus, an object positioned higher than 

15 the road surface, such as a roadside tree OB of FIG. 12B, is 
determined as an obstacle. Also, when an object is 
approaching from substantially right behind like a vehicle MM, 
a motion vector is obtained in the vicinity of the vanishing 
point, but its size is very small. Therefore, when the 

20 distance Z is obtained in the aforementioned manner, a value 
representing that the object is positioned below the road 
surface may be obtained. Since no object is generally 
present below the road surface, such a motion vector is 
determined as a moving body, so as to be processed through 

25 the moving body area extracting processing S24. 
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Through the aforementioned processing, an obstacle, a 
moving body, an approaching object and their distances in the 
image are obtained on the basis of the respective motion 
vectors of the points on the screen (S29), and the resultant 
information is output to the image synthesizing means HO. 

The image synthesizing means 110 synthesizes a frame 
203 of the rectangular area 202 to be lighted in red on the 
camera image input from the imaging means 101 as shown in FIG. 
13 and outputs the synthesized image to the display device 
111. The display device 111 displays an image obtained by 
laterally inverting the synthesized image so as to be in the 
same phase as an image on a rearview mirror. 

In this manner, the driver sees the display image as 
shown in FIG. 13 so as to recognize the approaching of the 
approaching object owing to the lighting of the red frame 203. 
Specifically, the driver sees the camera image so as to 
confirm the environmental conditions and can naturally pay 
attention to an approaching object that should be 
particularly careful about without being surprised by an 
alarm sound or the like. 

Also, information regarding from which direction and to 
what extent an approaching object is approaching to the self- 
vehicle, which is difficult to inform by using an alarm sound, 
can be directly and easily grasped. 

Although an approaching moving body alone is displayed 
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with a flashing red frame in this embodiment, a method for 
drawing attention of the driver is not limited to this but 
another color may be used or the red frame may be displayed 
without flashing. Furthermore, when the image of an 
approaching object displayed on the display device 111 is 
moving downward, it can be understood that the object is 
approaching the moving body, and when the image is moving 
upward, it can be understood that the object is moving away 
from the moving body. 

Moreover, since the distance to the approaching object 
is obtained, the grasp of the environmental conditions by the 
driver can be further supported by displaying the distance 
itself or changing the display in accordance with the 
distance. The color of the frame may be changed in 
accordance with the distance, for example, the frame is 
displayed in green when the distance is 50 m or more, in 
yellow when the distance is less than 50 m and 20 m or more 
and in red when the distance is less than 20 m, or the 
distance value itself may be displayed in the upper right 
portion of the moving body area. 
(EMBODIMENT 2) 

FIG. 14 is a block diagram for showing the structure of 
a moving body image display system serving as a driving 
support system according to Embodiment 2 of the invention. 
In FIG. 14, like reference numerals are used to refer to like 
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elements shown in FIG* 1 so as to omit the detailed 
description. Composing elements different from those shown 
in FIG. 1 are second imaging means 401 provided separately 
from first imaging means 101/ hierarchical block stereo- 
matching means 405, 3D information estimating/obstacle 
detecting means 409 and 3D image synthesizing means 410 
serving as image synthesizing means or image generating means. 
The composing elements 102 through 109, 405 and 409 together 
form detection means. 

An object of the moving body image display system of 
this embodiment is accurately detecting an approaching object 
or an obstacle close to a vanishing point having motion 
vectors (flows) difficult to accurately obtain. The 
operation of the moving body image display system of this 
embodiment will now be described with reference to FIGS - 15 
through 21. 

FIG. 15 is a schematic diagram of a self-vehicle and 
the surrounding area seen from above. As shown in FIG. 15 , 
in the driving support system of this embodiment, two imaging 
means 101 and 401 are installed on a rear portion of the 
vehicle apart from each other in the horizontal direction. 
The imaging means are disposed so as to have an overlap area 
OL where a first imaging range VA1 of the first imaging means 
101 serving as one imaging means and a second imaging range 
VA2 of the second imaging means 401 serving as another 
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imaging means overlap with each other. 

The imaging means are disposed as shown in FIG. 15 in 
order to take an image of and monitor the rear of the vehicle 
with a wide visual field by using a camera having a general 
lens with a limited visual angle. When the plural imaging 
means are thus disposed, an image with a large visual field 
can be taken without using a special lens such as a fisheye 
lens. Also, by using the plural imaging means, the 
resolution of the obtained image can be kept high. 

Moreover, it is necessary to combine obtained plural 
images together for display in the case where the system is 
used for monitoring. In the case where images taken from 
different positions as shown in FIG. 15 are combined, the two 
imaging ranges are preferably overlapped in consideration of 
occlusion accompanying the positions. 

The processing will now be described in detail. The 
hierarchical image generating means 102, the LPF 103, the 
block sampling means 104 and the hierarchical block matching 
means 105 carry out the same processing as that of Embodiment 
1 on each camera image input from the imaging means 101 and 
401, so as to obtain flows (motion vectors). 

FIGS. 16A and 16B are diagrams in which the flows 
obtained in the aforementioned manner are overlapped on the 
camera images obtained by the imaging means 101 and 401, 
respectively. OL denotes an overlap area between the camera 
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images. In FIGS. 16A and 16B, flows with respect to still 
objects such as a building and a tree are indicated by solid 
arrows, and flows with respect to moving bodies such as a car 
are indicated by broken arrows. 
5 The flows with respect to the still objects are flows 

directing toward the vanishing point VP owing to the movement 
of the self -vehicle. The size of each of these flows is in 
proportion to the speed of the self-vehicle and the distance 
to the vanishing point VP on the screen. Therefore, a flow 

10 in the vicinity of the vanishing point VP has a small size, 
and hence is difficult to detect. 

Therefore, the hierarchical block stereo-matching means 
405 carries out stereo-analysis of the overlap area OL of the 
two images of FIGS. 16A and 16B so as to obtain a stereo 

15 parallax. Since the vanishing point VP is formed right 
behind in the running direction of the vehicle, the imaging 
means 101 and 401 can be easily installed so that the overlap 
area OL includes the vanishing point VP in the camera image. 

FIG. 16C is a diagram obtained by overlapping the two 

20 images of FIGS. 16A and 16B. In this drawing, dislocation VD 
in the image of the vehicle corresponds to the obtained 
stereo parallax. 

When the imaging means are installed to be arranged in 
the horizontal direction , the stereo parallax VD is caused in 

25 substantially the horizontal direction. Therefore, an SAD (a 

26 



If: 



fii 



sum of absolute differences ) is obtained through block 
matching in a range of horizontal 5 pixels x vertical 3 
pixels from the image of the higher hierarchy, and a stereo 
parallax is obtained from a point where the SAD is minimum, 
5 With respect to the image of the lower hierarchy/ a stereo 
parallax is further obtained in a range of horizontal 5 
pixels x vertical 3 pixels with a stereo parallax obtained in 
a block in the same position in the image of the higher 
hierarchy as the center. 

10 The sub-pixel estimating/reliability determining means 

106 estimates, by using the motion vector obtained in the 
image of the lowest hierarchy (camera image) and the SAD, a 
motion vector at accuracy of one or less sub-pixel based on 
the SAD minimum point and SAD values at eight points around 

15 the SAD minimum point. Also, it determines the reliability 
of the motion vector in the block. 

Furthermore, the processing for the motion vectors is 
applied to the stereo parallax VD obtained by the 
hierarchical block stereo-matching means 405 in the 

20 completely same manner, so as to estimate a stereo parallax 
at accuracy of a sub-pixel and determine the reliability of 
the stereo parallax. 

Through the aforementioned processing, a flow 
corresponding to movement with time from a previous frame is 

25 obtained with respect to each point on the camera image, and 
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the stereo parallax is obtained in the overlap area 0L, 

In general, when the stereo parallax and the relative 
positional relationship between the imaging means are found, 
a distance from the imaging means to the object can be 
obtained through the principle of the trigonometrical survey. 
Also, on the basis of a flow corresponding to the movement 
with time of an image, by assuming that, for example, the 
object is still on the ground, the distance from the imaging 
means to the object can be obtained in relation to the 
running speed. 

By using this, the 3D information estimating/obstacle 
detecting means 409 estimates three-dimensional information 
with respect to the two camera images as shown in FIG. 17, 
and detects, as an obstacle OB, an area on the screen 
estimated to be at a predetermined height or more above the 
ground . 

air- 
AlSO, the moving body/approaching object detecting 

means 109 extracts, as a moving body /approaching object 

candidate block, a block having a motion vector different 

from a flow directing toward the vanishing point VP, and 

extracts a moving body MM as shown in FIG. 17 by linking 

moving body/approaching object candidate blocks close to one 

another. Furthermore, when it is determined that the 

distance to an obstacle present in an area in the vicinity of 

the vanishing point VP is approaching on the basis of the 
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three-dimensional information obtained from the 3D 
information estimating/obstacle detecting means 409, this 
object is extracted as an approaching object AP as shown in 
FIG. 17. 

As shown in FIG. 17, the 3D image synthesizing means 
410 utilizes the three-dimensional information obtained from 
the 3D information estimating/obstacle detecting means 409 so 
as to synthesize the two camera images input from the imaging 
means 101 and 401. Furthermore, it displays, on the 
synthesized image, areas of the moving body MM and the 
approaching object AP surrounded with, for example, lighting 
red frames, and displays the obstacle OB surrounded with, for 
example, a green frame, and they are output to the display 
device 111. At this point, the display device 111 displays 
an image obtained by laterally inverting the synthesized 
image to be in the same phase as an image on a rearview 
mirror . 

The driver sees the display image as shown in FIG. 17, 
so as to be informed of approaching of the moving body MM and 
the approaching object AP owing to the lighting of the red 
frames, and can directly and easily grasp from which 
direction and to what extent the approaching object AP is 
approaching the self-vehicle. Furthermore, the driver can 
directly and easily grasp the presence and the position of 
the obstacle owing to the display of the green frame. 

29 



S . 3 



Furthermore, as in the second conventional example and 
in Embodiment 1 of the invention, in the method so-called 
"motion stereo" for detecting an obstacle and an approaching 
object through movement analysis of a camera image, three- 
5 dimensional information of the camera image is obtained by 
analyzing change of an image in accordance with change of a 
visual point through movement. In this case, there is a 
problem that the change of an image taken in a moving 
direction or a reverse direction in accordance with the 
10 change of the visual point is small in an area in the 
vicinity of the vanishing point of flows (motion vectors) 
derived from the movement. When this method is applied to, 
for example, a vehicle, there arises a problem that the 
sensitivity of the detection is lowered in a region closer to 
15 the forward or backward region along the running direction. 

However, according to Embodiment 2, detection in an 
area in the vicinity of the vanishing point on the screen is 
compensated by the stereo analysis of the overlap area OL 
obtained by the two cameras, and hence, detection can be 
20 carried out at high sensitivity. 

The 3D image synthesizing means 410 utilizes the three- 
dimensional information obtained from the 3D information 
estimating/obstacle detecting means 409 so as to 
appropriately inform the driver of the information of the 
25 approaching object and the obstacle. The method will now be 
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described. 

In Embodiment 1, when a camera image synthesized with 
the information of an approaching object is seen, a distance 
in a depth direction in the screen should be determined based 
on the apparent size of the object on the synthesized image. 
In particular, when the imaging means is installed on a 
vehicle, it cannot be provided in a position higher than the 
height of the vehicle and should be directed substantially 
horizontal direction for allowing a rather far place to come 
in sight. As a result, the distance to an approaching object 
is unavoidably in the depth direction in the screen. 

As a method for easily grasping the distance to an 
approaching object on a screen, a technique to change the 
visual point of a synthesized image as disclosed in Japanese 
Patent Application No. 10-217261 devised by the present _ 
inventors can be employed. In a system disclosed in this 
application, plural camera images around a vehicle are 
assumed to be obtained by three-dimensionally taking images 
of the ground, so as to generate a synthesized image from a 
new visual point, for example, as if it were looked down from 
the sky. In this case, a distance between the vehicle and 
another object is in proportion to the distance on the screen, 
and hence, it can be intuitively grasped. 

The 3D image synthesizing means 410 of this embodiment 
utilizes the aforementioned technique to convert the camera 
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images obtained by the imaging means 101 and 401 into a 
synthesized image seen from a visual point higher than the 
actual installation positions of the imaging means. In order 
to attain a large visual field up to a far place, a visual 
5 point looking obliquely downward is preferred to a visual 
point looking right below. 

FIG. 18A shows the position of a virtual visual point 
WP for obtaining such a synthesized image. The virtual 
visual point WP is positioned above the imaging means 101 
10 and directed to an obliquely downward direction behind the 
vehicle. When the actual camera image obtained by the 
imaging means 101 is assumed to be at the default distance 
L, described with reference to FIG. 11A, the synthesized image 

laasr 

u seen from the virtual visual point WP can be generated from 

q 15 the actual camera image as shown in FIG. 18B. 

For example, when the actual camera image is as shown 
in FIG. 19A, the synthesized image seen from the virtual 
visual point WP is as shown in FIG. 19B. As is understood 
from FIG. 19, a white line 411 and the like actually present 
20 on the road surface at the default distance are synthesized 
in accurate positions also on the synthesized image, and 
since it is an image seen from above, the distance is 
enhanced to be easily grasped. However, a tree 412 and a 
vehicle 413 that are not actually present on the road surface 
25 are elongated to be unnaturally distorted on the synthesized 
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image. The synthesized image shown in FIG. 19B includes an 
area 414 outside the image actually taken. 

Specifically, when the conventional image synthesizing 
technique is directly applied, since the position of the 
ground alone is assumed in the image conversion, objects not 
actually present on the ground, such as another vehicle and 
an obstacle, are dis advantageously distorted on the 

synthesized image. 

Therefore, in this embodiment, the three-dimensional 
information obtained from the 3D information 
estimating/obstacle detecting means 409 is utilized, so that 
the distortion of a vehicle and the like caused on the 
synthesized image can be largely improved. This will be 
described with reference to FIGS. 20 and 21. 

in this embodiment, the three-dimensional information^ 
obtained from the 3D information estimating/obstacle 
detecting means 409 is utilized for detecting an obstacle and 
an approaching object present above the ground. Therefore, 
an area of the obstacle or the approaching object is 
synthesized in accordance with the three-dimensional 
information as shown in FIG. 20A, and hence, even when an 
image seen from the virtual visual point WP is synthesized, 
the synthesized image can be a natural image free from 
distortion. 

Specifically, with respect to the tree 412 and the 
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vehicle 413 actually not present on the road surface, the 
actual distances RD from the imaging means 101 are detected, 
and therefore, the areas are synthesized in positions 
determined in consideration of the actual distances RD on the 
5 synthesized image. Accordingly, as shown in FIG. 20B, the 
tree 412 and the vehicle 413 not actually present on the road 
surface are synthesized not on the basis of the default 
distances but on the basis of the actual distances RD, and 
hence, they are not elongated but can be natural on the 

10 synthesized image. 

In this manner, in the synthesized image, a white line 
and the like that are actually present on the road surface at 
the default distances are synthesized in accurate positions, 
and also with respect to a tree and a vehicle that are not 

15 actually present on the road surface, natural images can be 
obtained. Furthermore, since the synthesized image is an 
image seen from above, an image in which a distance can be 
easily grasped can be presented to the driver. 

FIG. 21 is a diagram of generation of a synthesized 

20 image from the camera images of the two imaging means 101 and 
401. As shown in FIG. 21A, also in this case, with respect 
to a tree 412 and a vehicle 413 that are not actually present 
on the road surface, the actual distances RD from the imaging 
means 101 and 401 are detected, and hence, the areas are 

25 synthesized in positions determined in consideration of the 
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actual distances RD on the synthesized image. Thus, a 
synthesized image as shown in FIG. 21B is obtained. In the 
synthesized image of FIG. 21B, VA1 and VA2 respectively 
denote areas corresponding to the visual field ranges of the 
5 imaging means 101 and 401 , and OL denotes an overlap area 
where the visual field ranges are overlapped. As compared 
with the image shown in FIG. 20B, an image with a wide visual 
angle is synthesized. Furthermore, in an area outside the 
overlap area OL, a distance can be also obtained based on a 

10 flow because it is away from the vanishing point, and 
therefore, for example, a tree 415 can be synthesized in a 
position in consideration of the distance. 
(MODIFICATION OF EMBODIMENT 2) 

A modification of Embodiment 2 will now be described 

15 with reference to FIGS. 22 and 23. 

FIG. 22 is a diagram for showing an example of the 
arrangement of the . imaging means according to this 
modification. Differently from the aforementioned embodiment, 
the two imaging means 101 and 401 are installed to be apart 

20 from each other at a distance k not in the horizontal 
direction but in the vertical direction. Also, with respect 
to the visual field ranges of the imaging means, similarly to 
Embodiment 2, the first imaging area VA1 of the imaging means 
101 and the second imaging area VA2 of the imaging means 401 

25 are overlapped with each other so as to have the overlap area 



OL. 

Similarly to Embodiment 2, a stereo-analysis area ST is 
provided in the overlap area between the two imaging areas, 
and movement analysis areas MAI and MA2 are provided in other 
5 portions* 

The processing in the stereo-analysis area ST of this 
modification will now be described. 

As shown in FIG. 23B, a stereo parallax VD is caused in 
the vertical direction in these two images. At this point, 

10 horizontal edges HE are previously extracted from the two 
images as shown in FIG. 23C, and the two images are compared 
with each other with respect to the horizontal edges HE, so 
as to easily obtain the stereo parallax VD. Furthermore, an 
image of a vehicle running on the road, which is the 

15 principal object of this invention, includes a larger number 
of horizontal edges, such as the bumper and a line of the 
bonnet, as compared with other edges, and hence can be easily 
detected. Since a distance from the imaging means is 
obtained from the stereo parallax VD, an approaching object 

20 and an obstacle can be detected in the same manner as in the 
aforementioned embodiment, and an image of an approaching 
object or an obstacle can be three-dimensionally synthesized 
to be displayed. 

In this modification, the two imaging means and the 

25 virtual visual point are arranged in the vertical direction, 



and hence, a vertical parallax is enhanced in the synthesized 
image. Furthermore, in an area where the number of 
horizontal edges is so small that the stereo parallax VD 
cannot be accurately obtained, the accuracy of the 
5 synthesized image is also lowered, but since the number of 
horizontal edges is small, unnaturalness caused by positional 
dislocation is minimally recognized. Accordingly, the 
synthesized image is very natural as a whole, and in an area 
where there are horizontal edges ( such as an area where 

10 another vehicle is imaged), a synthesized image in which 
information on a position in the depth direction is enhanced 
can be generated. 

Now, the relationship between the distance k between 
the imaging means and a detectable distance will be 

15 complementarily described. For simplifying the description, 
it is assumed that the imaging means is directed toward right 
behind. Assuming that an angle of the imaging means in the 
horizontal direction is approximately 90 degrees, when a 
distance D is 10 m, the field angle range in the horizontal 

20 direction corresponds to approximately 20 rtu Furthermore, 
when the aspect ratio of pixels of the imaging means is 3:4, 
the field angle range vh in the vertical direction 
corresponds to approximately 15 m. Specifically, Vh / D = 
1.5. Also, when the number of pixels in the vertical 

25 direction of the screen is Vp, a distance D required for a 



horizontal edge of a vehicle running behind and positioned at 
the same height as the imaging means to approach for causing 
a parallax corresponding to one pixel from parallax zero at 
infinite is obtained, on the basis of a formula, Vh / k = Vp, 

as follows: 

D=k*Vp/1.5 
When the number Vp of pixels in the vertical direction is 480 
and the distance k between the imaging means is 10 cm (= 0.1 
m), the distance D is roughly estimated as follows: 

D = 0.1 * 480 / 1.5 = 32 (m) 
The actual accuracy for detecting a parallax is 0.3 through 1 
pixel, and assuming that a desired distance for detecting an 
approaching object is 30 through 70 m and that the field 
angle of the imaging means is 60 through 120 degrees, the 
distance between the imaging means may be approximately _7_ 
through 20 cm according to rough calculation. 

In this manner, even when the imaging means are 
provided to be apart from each other at a comparatively small 
distance of approximately 7 through 20 cm, an approaching 
object can be detected, and hence, the system can be 
installed on a vehicle with more ease. 

A region behind a vehicle is described as a principal 
monitoring region in Embodiments 1 and 2, which does not 
limit the invention. For example, a region in front or on a 
side of the vehicle may be the monitoring region so as to 
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generate an image of the region in front or on the side of 
the vehicle. 
(EMBODIMENT 3) 

FIG. 24 is a block diagram for showing the structure of 
5 a driving support system according to Embodiment 3 of the 
invention. In FIG. 24, composing elements different from 
those shown in FIG. 14 are danger level determining means 501, 
external warning means 502 and passenger protecting means 503. 
Similarly to Embodiment 2, an overlap area between 

10 plural imaging means 101 and 401 is stereo-analyzed by the 
hierarchical stereo-matching means 405, so as to accurately 
detect an approaching object right behind the vehicle, which 
is conventionally difficult to detect. 

A conventional system for detecting an approaching 

15 object does not cover an object approaching from right behind 
not only because such an approaching object is difficult to 
detect merely by movement detection but also because there 
are few methods for avoiding collision or the like after the 
detection of such an approaching object. For example, in the 

20 first conventional example, in the case where an object 
approaching from a lane adjacent to the lane where the self- 
vehicle is running is detected, collision can be easily 
avoided by warning not to change the course to the adjacent 
lane. On the other hand, with respect to a vehicle 

25 approaching in the same lane as the self -vehicle, it is 



necessary to make positive movement, such as to evade forward 
by increasing the speed or to change the course to another 
lane. In either case, there arises another risk such as 
collision with another vehicle or object other than the 
5 approaching object. 

Therefore, in this embodiment, when the moving 
body/approaching object detecting means 109 detects an object 
approaching from right behind, the danger level determining 
means 501 determines a possibility of collision with this 

3 10 approaching object on the basis of time up to the collision, 

Irii 

00 change of the speed of the approaching object and the like. 

*0 when it determines that there is a strong possibility of 

L collision, an indicator signal is output. The external 

u warning means 502 receives the indicator signal and gives a 

h! 15 warning behind by, for example, automatically lighting a_ 
brake light. This warning is not limited to the lighting of 
a brake light but may be radiation or flash of another light 
provided to face backward, an alarm sound, or a warning using 
electric waves such as a radio. 
20 When the danger level determining means 501 outputs the 

indicator signal, the passenger protecting means 503 takes 
measure to protect passengers such as winding up a seat belt 
or preparation for the operation of an air bag system. With 
respect to the air bag system, if it is previously found that 
25 there is a strong possibility of collision, various 
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preparation such as the preparation for the operation and 
detection of heads of passengers can be carried out, so as to 
definitely protect the passengers. 

The detection means for an approaching object of this 
embodiment is not limited to the means utilizing the stereo- 
images and the movement analysis but may be another means 
using, for example, radar or laser. 

It goes without saying that the invention can be easily 
applied to a moving body other than a vehicle, such as a ship, 
an airplane and a train. 

Also, the installation positions and the number of 
plural imaging means are not limited to those described 
herein. 

Furthermore, the functions of the detection means and 
the image generating means of the driving support system of_ 
this invention may be wholly or partially realized by 
dedicated hardware or software. Also, a recording medium or 
a transfer medium storing a program for making a computer 
execute the whole or a part of the functions of the detection 
means and the image generating means of the driving support 
system of this invention may be used. 

As is obvious from the above description, according to 
the invention, an approaching object can be detected without 
being influenced by vibration derived from jolt of a vehicle. 
Also, when the stereo-analysis using a plurality of imaging 
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means is utilized together, an approaching object right 
behind that causes a small movement change in an image can be 
detected • 

Since the detection result of an approaching object and 
5 the like is informed on a display image, the driver can 
directly confirm the positional relationship and the 
environmental conditions • Furthermore, when an image is 
converted into an image seen from a higher visual point 
looking obliquely downward, the distance to an approaching 
10 object can be displayed more simply. 

Moreover, not only the driver is informed but also a 
warning is given to an approaching vehicle, the possibility 
of collision can be lowered, and measure to protect 
passengers from the impact of the collision can be started 
15 earlier. 

In this manner, the present invention provides a 
driving support system for reducing burdens of a driver. 
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CLAIMS 

1. A system for supporting driving of a moving body 
comprising: 

a plurality of imaging means installed on said moving 
body for taking camera images of the rear of said moving 
body; and 

detection means for detecting movement of an object 
present in the rear of said moving body based on said camera 
images of said plurality of imaging means , 

wherein said plurality of imaging means have, in camera 
images thereof , an overlap area that includes an area in the 
vicinity of a vanishing point and in which imaging areas of 
first imaging means and second imaging means overlap with 

each other, and 

said detection means obtains a stereo parallax between 
said first imaging means and said second imaging means in 
said overlap area, and obtains a distance to said object on 
the basis of said obtained stereo parallax. 

2. The driving supporting system of Claim 1, 

wherein said detection means detects flows 
corresponding to movement with time of an image in the 
imaging area of said first imaging means excluding said 
overlap area, and detects movement of said object present in 
the rear of said moving body on the basis of said detected 
flows . 
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3. The driving supporting system of Claim 1, further 
comprising image synthesizing means for generating an image 
representing the rear of said moving body through image 
synthesis using said camera images of said plurality of 
imaging means. 

4. The driving supporting system of Claim 1, further 
comprising: 

danger level determining means for determining a 
possibility of collision of said moving body with an 
approaching object from the rear of said moving body on the 
basis of information output from said detection means and for 
outputting an indicator signal when it is determined that 
there is a strong possibility of the collision; and 

external warning means for providing a warning to the 
rear of said moving body when said indicator signal is output 
from said danger level determining means. 

5. The driving supporting system of Claim 1, further 
comprising: 

danger level determining means for determining a 
possibility of collision of said moving body with an 
approaching object from the rear of said moving body on the 
basis of information output from said detection means and for 
outputting an indicator signal when it is determined that 
there is a strong possibility of the collision; and 

passenger protecting means for taking measure to 
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protect a passenger of said moving body when said indicator 
signal is output from said danger level determining means. 

6. A system for supporting driving of a moving body 
comprising: 

imaging means installed on said moving body for taking 
a camera image of a surrounding region of said moving body; 

image generating means for converting said camera image 
of said imaging means into an image seen from a visual point 
positioned differently from said imaging means; and 

detecting means for detecting a distance from said 
moving body to an object imaged in said camera image, 

wherein said image generating means corrects distortion 
of an image of said object by using said distance detected by 
said detection means in converting said image. 

7. The driving supporting system of Claim 6, 
wherein said imaging means is plural in number, 

said plural imaging means have, in camera images 
thereof, an overlap area in which imaging areas of first 
imaging means and second imaging means overlap with each 
other, and 

said detection means obtains a stereo parallax between 
said first imaging means and said second imaging means in 
said overlap area, and obtains a distance to said object on 
the basis of said obtained stereo parallax. 

8. The system for supporting driving of a moving body 
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of Claim 6, 

wherein said detection means obtains a distance to said 
object on the basis of flows corresponding to movement with 
time of said camera image. 

9- A system for supporting driving of a moving body 
comprising: 

imaging means installed on said moving body for taking 
a camera image of a surrounding region of said moving body; 
and 

detection means for obtaining flows corresponding to 
movement with time on the basis of said camera image of said 
imaging means and for detecting movement of an object present 
in the surrounding region of said moving body on the basis of 
said flows, 

wherein said detection means obtains, as preparation 
for detecting the movement of said object, an offset 
estimated value from each of said obtained flows and cancels 
said offset estimated value from each of said flows as a 
vibration component derived from jolt of said moving body. 
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Abstract 

A driving support system for enabling a driver to 
appropriately confirm environmental conditions of a moving 
body is provided. A plurality of imaging means for taking an 
5 image of the rear of the moving body is installed on the 
moving body. The plurality of imaging means have an area 
(0L) where their imaging areas overlap with each other , and 
the overlap area (OL) includes an area in the vicinity of a 
vanishing point (VP). Detection means obtains a stereo 
10 parallax (VD) between one imaging means and another imaging 
means in the overlap area (OL), and obtains a distance to an 
object based on the stereo parallax (VD). 
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disclosed in the prior United States or PCT International application in the manner provided 
by the first paragraph of 35 U.S.C. Section 112, I acknowledge the duty to disclose to the 
United States Patent and Trademark Office all information known to me to be material to 
patentability as defined in Title 37, C.F.R., Section 1.56 which became available between 
the filing date of the prior application and the national or PCT International filing da te of this 
application: 

Application Serial No. Filing Date Status - patented, 

pending, abandoned 



I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, under Section 1 001 of Title 1 8 of the 
United States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 
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POWER OF ATTORNEY: As a named inventor, I hereby appoint the following attorney(s) 
and/or agent(s) to prosecute this application and transact all business in the Patent and 
Trademark Office connected therewith, (list name and registration number) 



Send Correspondence to: 



Customer No. 27572 

Harness, Dickey & Pierce, P.L.C. 



Direct Telephone Calls to: (name and telephone number) 

\^ GregoryJL Stobbs (248) 641-1600 



W Full name of sole or first inven t o r H irofumi I SHII 



run name ui owi^ wi . . ~. ^y.- — t 

Inventor's signature: _ 

w 



- , November 30, 2001 

Date- 



Residence: K g na g a wa, .Ja pan 
Citizenship: Japan 



Post Office Address: fi-fi-17-910, shimn-nHanaka , Nakahara-kn Kawasaki -shi , Kanaqawa 
01 1 -0041 J Japan . — . 



Full name of second joint inventor, if any ^JU n inh i M O ni M I I R A 
Second Inventor's signature: i_ 



Date . November 30, 2001 



Residence: 
Citizenship: Japan 




Post Office Address: a-14-ft Ni s hi to mi qaoka N ara-shi Nara 631 -0006, Japan 



Page 4 of 

Full name of third joint inventor, if a£)^y: _J\/la ^a mir hi M AK AGAW^_^ 

Third Inventor's signature: %»&toticli ^cX^Omt^ 

p Rt *- November ^0. 2001 . 

Residence: _Qsaka*Japan_ 

Citizenship: Japan — — - 



Post Office Address: 22^5=304, Euisakazkit amacfai, Hirak ata-shi . Osaka 573-0151 , Japan 



Full name of fourth joint inventor, if a£Ty^_Shj isakj ■ okamoto^ 



Fourth Inventor's signature: 



Date: I\ T o verier* ^0 f 2001 



Residence: _P sakar T 7ta P an L - 




Citizenship: Japan 



Post Office Address: a-fi-m-Pis , Mn^higanka , Hirakata-shi Osaka 573-0013 Japan 



Full name of fifth joint inventor, if any: 
Fifth Inventor's signature: 



